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Higher education staff involved in e-learning often struggle with organising their student support 
activities. To a large extent this is due to the high workload involved with such activities. We distin- 
guish support related to learning content, learning processes and student products. At two different 
educational institutions, surveys were conducted to identify the most critical support activities, 
using the Nominal Group Method. The results are discussed and brought to bear on the distinction 
between content-related, process-related and product-related support activities. 


Introduction 

Modem higher education curricula increasingly make use of information and 
communication technologies, and thus increasingly implement e-learning. This 
applies to open, distance education as well as more traditional forms of education 
(Guri-Rosenbilt, 2005). It is well Imown that the introduction of e-leaming often 
leads to an increase in staff time spent on tutoring (Bartolic-Zlomislic & Bates, 1999j 
de Bie, 2002; Romiszowski & Ravitz, 1997 as cited in Fox & MacKeogh, 2003). 
Rumble (2001, pp. 81-82) suggests that there is even a doubling of the workload. 
One of the most important reasons for this is that often an extended classroom model 
is applied (Beaudoin, 1990j Salmon, 2004). That is, in addition to the usual lectures 
and availability during office hours, a teacher creates a web site to support the course 
and is available for email help between classes. Staff who use e-learning environments 
to organise student support this way profit little from economy of scales: students are 
treated as help seeking individuals, rather than as a group. The problem is further 


* Corresponding author. Educational Technology Expertise Centre, Open University of the 
Netherlands, P.O. Box 2960, NL-6401DL Heerlen. The Netherlands. Email: fred.devries@ou.nl 

ISSN 0968-7769 (print)/ISSN 1741-1629 (online)/05/030219-l 1 
© 2005 Association for Learning Technology 
DOI: 10.1080/09687760500376488 


220 F. J. de Vries et al. 


exacerbated, as online students — rightly or wrongly — expect to be able to send emails 
to support staff and receive answers quickly (Salmon, 2004) . 

In addition to this the characteristics of the questions for support change. As 
Anderson (2004) points out, tutors in an e-leaming context are no longer restricted 
to well-defined and pre-planned tasks but have to adapt on the fly to student needs. 
Tutors have to make provisions for negotiation of their activities to meet unique 
learning needs, and equally well have to stimulate, guide and support the learning in 
a way that responds to common and unique student needs. 

A way out of this predicament is to develop technologies that make support activi- 
ties significantly less time consuming. This is not as easy as it may sound. Although 
it is not hard to incorporate technology in teaching and learning, this does not neces- 
sarily lead to more efficient practices. Furthermore, it is also important to determine 
first whether there is a real user need for software to be developed (Cooper & Saun- 
ders, 2000) . This kind of approach readily becomes costly, more time-consuming and 
thus ineffective. So the challenge is to select e-learning-based support activities that 
profit from technology support and to make sure that the perceived need is a genuine 
one (Koper, 2004). 

Three categories of support activities in blended learning environments are 
distinguished: support related to the learning content (CONT), support related to 
the learning process (PROC) and support related to the learning product (PROD) 
(Reid & Newhouse, 2004). Content-related support refers to all tutor activities that 
pertain to the subject matter. Cases in point are answering content-related ques- 
tions, providing additional explanations or examples with regard to the subject 
matter. As a result of the introduction of e-leaming, staff report an increase in the 
flow of content questions and answers from and to students. This is partly due to 
an increase in opportunities to reflect as a consequence of written interaction 
instead of oral interaction with the result that a content question is discussed on a 
deeper level. In addition, the existence of an interaction pattern that involves all 
students instead of only a few of them also intensifies the tutor-student communi- 
cation. After all, a question posed by the staff to enable students to rehearse the 
subject matter now reaches each student individually and not only those who pay 
attention (Coppola et al., 2002). 

Process-related support refers to all tutor activities related to the learning process 
of individual learners or group collaboration. Examples include providing ‘study aids’ 
or moderating group discussions. In blended learning environments, staff now see 
that their role is changing from instructor to facilitator. Instead of being responsible 
for student acquisition of knowledge, their responsibility shifts to moderating student 
activity in, for example, collaborative groups (Coppola et al., 2002). In other words, 
due to the introduction of e-leaming the emphasis of the support activities shifts from 
content-related support (i.e. transmission of knowledge) to process-related support 
(i.e. facilitating and guiding students) (Beaudoin, 1990; Hardless & Nulden, 1999). 
As Denis et al. (2004 ) claim, an e-tutor is ‘... someone who interacts directly with 
learners to support their learning process when they are separated from the tutor in 
time and place for some or all these direct interactions’. 
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Product-related support refers to all tutor activities that pertain to the summative 
assessment of student products; such as checking the authenticity of the product or 
correcting tests. Although the emphasis on process-related support in blended learn- 
ing environments also results in a greater emphasis on formative assessment (Hard- 
less & Nulden, 1999), the summative assessment of student products remains an 
important support activity (Beaudoin, 1990). Plagiarism is particularly problematic 
in e-learning. It appears that the detection rates of plagiarism are only 1.5% and that 
approximately 20% of the tutors ignore obvious plagiarism because of the hassle of 
dealing with it (Bennett, 2005). It would seem, then, that extra effort is necessary to 
put a stop to these practices. 

Three categories of support activities — content-related, process-related and prod- 
uct-related support — have been discussed and illustrated by the changes that tutors 
have to face when e-learning is introduced in their curriculum. The present study 
aims to identify the critical support activities in both distance and e-learning-enriched 
education that can be supported by technical solutions. With regard to the activities 
identified, it furthermore investigates whether tutors in distance education identify 
problems that are different from those identified by their colleagues in traditional 
education. Finally, it explores whether the categories of support activities differ in 
perceived importance. Here too, the opinions of tutors in distance education are 
contrasted to those of tutors in traditional education. 


Materials and methods 

Two separate brainstorm sessions were organised, each structured according to the 
Nominal Group Approach (Dunham, 1998). In the Nominal Group Approach, after 
the topic has been presented to session participants, they are asked to take a few 
minutes to think about and write down their responses. Once everyone has given a 
response, participants are asked for a second or third response, until all of their 
answers have been noted. Once duplications are eliminated, each response is assigned 
a letter or number. Session participants are then asked to choose up to 10 responses 
that they feel are the most important and rank them according to their relative impor- 
tance. These rankings are collected from all participants and aggregated. 

One brainstorm session involved a group of stakeholders at an open distance learn- 
ing institute (Open University of The Netherlands (OUNL)), the other stakeholders 
were from a traditional teacher training institution (Fontys University for Professional 
Education, Fontys, The Netherlands). The latter have been practising forms of 
blended learning for several years. The participant groups at the two institutions did 
not interact with one another. 

Both institutions are situated in The Netherlands; but they cater for different 
students. The OUNL serves mainly students who study at their own pace and in their 
own time. Some are degree students, but most are lifelong learners who are studying 
to improve their job qualifications or just for pleasure. Students range in age from 18 
up to 80 and older. The OUNL relies on information and communications technol- 
ogy tools for its learning management, such as newsgroups and an in-house-built 
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virtual learning environment (VLE). The Fontys teacher training institute teaches 
full-time (day-time) as well as part-time students. The full-timers outnumber the 
part-timers only by a margin. For both groups, e-leaming tools such as e-mail, the 
Web and a VLF are used intensely. 

The OUNL group consisted of 1 2 people, including faculty members, educational 
designers and staff from the teacher training institute. They were not randomly drawn 
from the OUNL’s faculty, but rather carefully selected for their expertise and experi- 
ence, which included course design, tutoring, help-desk support, educational 
research and software development. Here three of the participants provided no scores 
at all or invalid scores and were subsequently ignored, bringing the OUNL group 
down to nine members. The Fontys group consisted of seven people, all working at 
the teacher training faculty. They were not chosen randomly either, they were 
selected because they fulfilled managerial duties beyond their teaching responsibili- 
ties. Not only would the group including all teachers have been too large, it was felt 
that teachers with managerial duties would have a more informed opinion on the 
matter. Two of the participants provided invalid scores and were subsequently 
ignored, bringing the Fontys group down to five members. 

By way of preparation, the organisers of the brainstorm invited the participants in 
each group to consider the following questions: 

• Which support activities currently lead to staff worldoad problems? 

• Which support activities do you find relevant but are not common practice because 

of workload constraints? 

At the subsequent face-to-face sessions (a separate one for each group), the brain- 
storm organisers (the same people for each group) briefly introduced the nominal 
group approach to the participants. Referring to the preparatory questions, every 
participant was then asked to draw on his or her personal experiences and describe as 
many situations as possible of critical student-support activities. They were stimu- 
lated to be creative and take risks; that is, also to take into consideration situations 
they believed to be critical but had not experienced so far. The participants within a 
group were stimulated to interact with each other, if they so wanted. However, they 
were asked not to criticise any contribution — all ideas were to be considered equally 
good or bad. 

With a great many ideas on display, the participants were then asked, as far as 
necessary, to briefly explain them. After eliminating the apparently synonymous items 
(this was discussed in the group) they were asked to add new ideas, if they felt they 
had any. At the end, the OUNL group came up with 37 items and the Fontys group 
with 13. 

Finally, the participants were asked to give votes to the items, as a reflection of an 
item’s degree of importance. They could attribute either 1, 2, ... 9 or 10 votes. More- 
over, each number of votes could be allocated once only. This means that, for each 
individual participant, 1 0 items would each receive a unique number of votes and the 
remaining would receive none. It also means that each participant had a total of 55 
votes to distribute. 
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The method followed is in accordance with the Nominal Group Approach. Clearly, 
another allocation scheme could have been followed; for instance, one in which 
participants could freely distribute votes over categories. In principle, they then could 
have allocated all votes to one item only or, conversely, have distributed their votes 
evenly over all items. We chose the present scheme with limited distribution possibil- 
ities as we wanted to force the participants to consider a sizeable number of items 
(10), but not so large a number that they could effectively refrain from choosing. The 
latter could happen easily, as the participants themselves came up with the various 
items and might thus be inclined to favour their own items over those of others. 

An effort was made to categorise the variety of items that resulted from the Nomi- 
nal Group Approach. Four experts, all authors of this paper, allotted each of the 50 
items to only one of three categories (CONT, PROG, PROD). After the first round 
of allotment, no effort was made to align the opinions of the experts. The language of 
the items about which the experts were ambiguous clearly was to blame, rather than 
confusion among the experts about the categories themselves. In order to estimate the 
inter-observer reliability, Cohen’s (1960) kappa value was used. The computation is 
straightforward for two observers (and several categories) or several observers and 
two categories. The present case has four observers and three categories. Following 
Fleiss (1981) and Landis and Koch (1977), three separate kappa values were first 
computed, one for each of the three categories (case several observers and two cate- 
gories). Effectively, for the computation of each individual kappa value, one of the 
three categories was set apart and the remaining two categories were grouped. These 
three kappa values provided insight in the inter-observer reliability per category. 
Subsequently, the overall kappa value was computed as the weighted average of the 
resulting kappa values (again, following Bonnardel). This provides insight in the 
inter-observer reliability for the entire allotment exercise. 


Findings 

Tables 1 and 2 present the items that came out of the nominal group session for the 
OUNL and Fontys groups. In both tables, each entry lists an item suggested by one 
or more of the participants. Votes reflect the cumulative votes of all the participants. 
Items are ordered by the number of votes they received. The material and methods 
section explains how the totals have been computed. Also the total number of votes 
each item received is indicated. This number (‘score’) is an indication of how relevant 
the participants deemed a particular item to be. 


Are all items equally important? 

The first question to be answered, for each institution separately, is whether the 
participants consider some items more important than others. Inspection of the 
tables suggests clear agreement on the side of the participants on what items matter 
and what do not. The possible range for the scores on each item in Table 1 is 0-90 
votes; in Table 2 it is 0-50 votes (the maximum score is the maximum score per 
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Table 1 . Critical situations as identified by selected staff members of the OUNL 


ID 

Support activities OUNL brainstorm 

Score 

Category 
(explained 
in the text) 

7 

Coaching of novice teachers: clarifying problems, generating alternatives 

0 

★ 

11 

Facilitate putting together groups with specific characteristics and demands 

0 

PROG 

24 

Monitor complex group processes 

0 

PROG 

27 

Monitor study behaviour and visibility of students in online learning 

0 

PROG 


environment 



35 

Score assignments and papers 

0 

PROG 

36 

Support for writing of papers 

3 

PROD 

6 

Coach group activities so everyone will participate 

4 

★ 

21 

Involve context of the student in feedback or answer (full-time student, 
part-time student, preliminary education, profile in portfolio) 

4 

CONT 

31 

Organise discussions in threads 

4 

PROG 

37 

Train students on the fly how to work with a VLE 

4 

★ 

2 

An avatar which will act in a group on behalf of a student 

6 

PROG 

28 

Monitor study progress, make interventions 

7 

PROG 

10 

Explanation of problems with re-usability (faq) 

8 

PROG 

25 

Monitor progress, prevent drop-out 

8 

PROG 

17 

Give feedback on the results of assignments 

9 

CONT 

23 

Management and archiving of discussion threads in groups 

10 

★ 

34 

Representation of student and tutor in community 

10 

PROG 

15 

Generate tests on the basis of learning objectives 

12 

PROG 

26 

Monitor study behaviour and compare with norms and past- performance 
(study pace and pressure) 

12 

PROG 

29 

Monitor study progress, personalise feedback 

12 

PROG 

1 

Tutor develops additional learning materials for individual students with 
specific needs and is supported in designing and editing these materials 

15 

CONT 

9 

Detect plagiarism by comparing papers sent in by students 

15 

PROD 

30 

Ordering of interactions 

15 

PROG 

3 

Automatic individualisation of learning materials and support services on 
the basis of student characteristics (progress) 

16 

★ 

33 

Present tutor commitment (via presence tools) 

16 

PROG 

22 

Just-in-time information for novice teachers, good examples, tips and tricks 

18 

CONT 

5 

Check texts on formal characteristics (length, style, references) and core 
concepts 

19 

PROD 

19 

Help find similar fellow-students 

19 

CONT 

32 

Present just-in-time information 

19 

PROG 

4 

Categorise questions and problems 

20 

PROD 

8 

Design a faq., on the basis of which questions are generated 

21 

PROD 

18 

Give insight in group activities (quantity, quality, frequency) by graphical. 

25 

CONT 


means 



20 

Intelligent, personalised frequently asked questions list 

25 

CONT 

13 

Filter out questions sent in by email and automatically answer these 

27 

PROG 

14 

Fraud in papers 

31 

PROG 

16 

Give feedback on the progress of students’ work 

32 

PROG 

12 

Filter out ‘repeated’ questions 

49 

PROG 


Total number of votes 

495 
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Table 2. Critical situations as identified by selected staff members of the Fontys Teacher Training 

School, Sittard 


ID 

Support activities Fontys brainstorm 

Score 

Category 
(explained 
in the text) 

43 

How to help students who are in the final phase of their study organise their 
own support activities 

1 

PROC 

50 

The administrative systems need to support the existing diversity of 
different training programmes 

7 

PROC 

42 

Give support in laboratories while students are working and walking around 

9 


48 

Individual problems can be clustered in order to provide support in a group 
session. These should be traceable both individually or for a whole group 

13 

PROC 

40 

Staff should limit the number of visits to their placement students by using 
efficient technical solutions like video observation and feedback via a VLE 

18 

PROD 

46 

Students can communicate using IT about their own questions and 
problems, without intervention of staff 

18 

PROC 

38 

Individual students might be helped in their preparation of lessons (being 
teacher-trainees) with a database of video materials, with explanation and 
well-designed assignments 

24 

CONT 

39 

Support systems should be organised so that requests for support from 
students can be met adequately 

24 

PROC 

49 

The demand for support is rising due to the individual learning 
arrangements; therefore, staff time needs to be organised more efficiently 

27 

PROC 

47 

Individual student’s products can be shared by a group, fellow-students can 
comment (electronically) 

28 

PROC 

45 

Just-in-time support on the basis of questions from students is growing; 
staff availability needs to be organised cleverly 

32 

★ 

44 

As students are expected to study independently, staff often receive no 
questions at all or too late; so staff cannot keep track of the study progress 

33 

PROC 

41 

Portfolios of students should be easily accessible and well structured, so that 
staff can easily find what they are looking for and do not lose much time 

41 

PROD 


Total number of votes 

275 



participant (10) multiplied by the number of participants (nine and five, respec- 
tively)). The actual ranges are 0-49 and 0-41. 

Closer inspection of the data in Tables 1 and 2 reveals that some items are deemed 
highly relevant: 

• fraud in papers (ID 14); 

• give feedback on the progress of students’ work (ID 1 6); 

• filter out ‘repeated’ questions (ID 12); and 

• portfolios of students should be easily accessible and well structured, so that staff 
can easily find what they are looking for and do not lose much time (ID 41); 

whereas others are seen as less relevant: 

• coach novice teachers: clarifying problems, generating alternatives (ID 7); 

• facilitate putting together groups with specific characteristics and demands (ID 11); 
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• monitor complex group processes (ID 24); 

• score assignments and papers (ID 27); 

• support for writing of papers (ID 35); and 

• how to help students who are in the final phase of their study organise their own 
support activities (ID 43). 

Looking at the results this way, little in the way of a pattern may be discerned. 
Although is it very useful to know that ‘the prevention of fraud with student papers’ 
or of ‘repeatedly having to answer a similar question’ are high on the agenda of items 
to be resolved, one would have liked to know what support issue in general staff at 
either institution are concerned with. Also, it would be interesting to Imow whether 
the institutions differ in what matters to them and what does not. Unfortunately, the 
nominal group method is ill-suited to answer this Idnd of question. As discussed, 
according to it, participants themselves formulate items. This has the benefit that one 
maximally taps into the creativity of the participants. It has the drawback that it is 
impossible to establish any trend in the participants’ judgements^ nor can one make 
intra-group comparisons. 

Do the opinions of participants within and between institutions concur? 

In order to be able to draw more general conclusions and to compare institutions, first 
each of the items produced by either group of participants was categorised by experts 
as being an instance of either content-related support (CONT), learning process- 
related support (PROC) or student product-related support (PROD) (see the final 
column in Tables 1 and 2). This way, a common denominator was established. The 
four experts received the following instruction when logging their category choices: 

• Content-related support refers to all those activities support staff undertake that 
are related to the subject matterj for example, answering content-related questions, 
providing additional explanations or examples with regard to the subject matter, 
and so on. 

• Process-related support refers to all those activities support staff undertake that are 
related to the learning process of individual learners or collaborating groups; for 
example, providing ‘study guides’ or moderating group discussions. 

• Product-related support refers to all those activities support staff undertake that are 
related to the formative assessment of student products; for example, checking for 
authenticity of the product or correcting tests. 

The subsequent analysis will bear on the categories and the number of items 
(frequencies) that have been allotted to them. The judgements of the experts that 
have carried out the allotment is a decisive factor. If the experts disagree on the cate- 
gories to which the items belong, further analysis is of little use. Fortunately, the 
weighted mean value of Cohen’s kappa equals 0.61, which according to Fleiss (1981) 
reflects a ‘good’ degree of inter-observer reliability. 

With the expert judgements, items were categorised according to the following 
rules: 
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Table 3. Critical situations categorised: absolute frequencies of items for the categories content, 

process and product 


Category 

OUNL staff 

Fontys staff 

Total 

CONT 

7 

1 

8 

PROC 

20 

8 

28 

PROD 

5 

2 

7 


• Items that according to three or four of the experts should belong to, say, category 
A were allotted to category A. Items to which this applies are labelled CONT, 
PROC, or PROD in Tables 1 and 2. 

• Items that less than three of the experts allotted to the same category were ignored. 
So all permutations of scores (2,2,0) and (2,1,1) were left out. In Tables 1 and 2 
these items are labelled with one or two asterisks, respectively. These seven items 
are then ignored, leaving a total of 43 items for further analysis. 

Table 3 presents the number of items that ended up in the different categories. A chi- 
square test for deviations from a uniform distribution of items over categories, which 
one would expect under the null-hypothesis of no prevalence, was not appropriate. 
The number of categories (three) is too small to convey any power on such a test. The 
test is further comprised by the low frequencies in two of the cells (1,2). Similar argu- 
ments go for a test of independence, which could show Fontys staff to hold a different 
opinion to OUNL staff. The table nevertheless suggests that both institutions believe 
that the process category is the most pressing, as it contains by far the most issues. 


Discussion and conclusions 

The use of the Nominal Group Method led to a bewildering diversity of critical situ- 
ations, identified by staff at both institutions. It did become clear, however, that 
collectively staff regarded some of the situations as important and others as not 
important. Tables 1 and 2 pointed to the particular importance of: 

• Preventing fraud in papers. 

• Giving feedback on the progress of students’ work. 

• Filtering out ‘repeated’ questions. 

• Making portfolios of students easily accessible and well structured. 

It is remarkable that three out of four are process-related support activities, prevent- 
ing fraud being the exception (it is a product-related activity). The suggested impor- 
tance of the process category is further reinforced by the observation that the four 
activities directly below preventing fraud are all process related (see Table 2). Table 
3 further underscores this finding. It shows that at both institutions the overwhelming 
majority of critical situations belonged to the process category. We may therefore 
conclude that staff consider process-related support activities most critical. As both 
institutions surveyed are in transit from traditional forms of distance learning and 
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face-to-face learning to their e-learning enhanced pendants, this conclusion is fully in 
line with the early reported literature finding that, upon the introduction of e-leam- 
ing, the emphasis shifts from content-related support to process-related support 
(Beaudoin, 1990; Hardless & Nulden, 1999). The high score of the fraud item is also 
in line with literature findings (Bennett, 2005). 

We need to be cautious of eliminating the apparently unimportant items and cate- 
gories. First, the method followed makes it hard to distinguish between items that are 
found genuinely unimportant and those that are formulated unattractively. The kind 
of interaction process in which participants of the sessions are engaged leaves little 
time for reflection and careful consideration. At least this implies that some items will 
attract few votes merely because of their opaque language or because they are not well 
thought through. Clearly, lack of votes does not reflect lack of importance, then, but, 
perhaps, difficulty to exactly pinpoint what the issue amounts to. 

Second, an argument may be made that process problems are first-order problems 
in that they screen off other problems, particularly content-related problems. A tran- 
sition to e-leaming, whether in a distance learning environment or more traditional 
learning setting, is fraught with difficulties — organisational, technical, pedagogical 
(cf. Sloep et al., in press). Inevitably, students will experience some of these too. 
These difficulties, irrespective of whether they pertain to content, process or product, 
will primarily make themselves felt in the learning process. After all, it is while 
learning that students see themselves confronted with the imperfections of their 
learning environment. The argument thus is that staff are indeed confronted with 
process issues, but that this is at least in part a reflection of the transition process that 
institutions are going through. 

In summary, then, when making the transition from traditional forms of learning 
to e-learning-enhanced, blended forms of learning, it is important to pay particular 
attention to process-related demands for support as well as fraud prevention. 
However, content-related and product-related support issues should not be ignored. 
They might well resurface once the transition nears completion and process issues 
have been resolved. 
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